After completing this lesson, you'll be able to:
In this lesson, you will:
After completing this lesson, you'll be able to:
In this lesson, you will:
If you're taking a live Safe Software-hosted training course or using an on-demand lab for this course, we've already imported the content for this lesson into your FME Flow.
If you're completing this lesson with your own FME Flow, you must import the starting project with the content into your FME Flow.
See FME Flow Authoring: Instructions to Import the Starting Project for directions to import the project to your own FME Flow.
In FME Flow Automations, you may split your workflows to process multiple actions simultaneously and use FME Flow's parallel processing capabilities with numerous engines. Splitting your workflow lowers the processing time by running multiple workspaces at once. It also increases fault tolerance, as workspaces that run in parallel will still process even if one workspace fails.
When you split your workflows, each stream creates an output. To combine the outputs into a single message in Automations, you use a Split-Merge Block. The Split-Merge Block lets you split the Automation workflow into multiple streams and then wait for all components within the block to complete before continuing the workflow along a single stream.
In this workflow, the trigger passes one message to the Split-Merge Block. The workflow is split and sends two messages, one to each workspace. Once the workspaces are complete, each workspace sends a message to the block's success port. The Split-Merge Block combines the two messages into one message and passes it along to the Validation workspace action. Two messages would pass to the Validation workspace without the Split-Merge Block, triggering it to run twice.
The Merge Messages action in FME Flow Automations functions similarily to the Split-Merge Block by merging messages and waiting to process all downstream actions until it receives messages from all previous actions. Compared to the Split-Merge Block, the Merge Messages action limits you from merging messages from multiple triggers and from Automation Writer outputs. Due to these, the Merge Messages action is legacy functionality in Automations and is only present to support backwards compatibility for your workflows. For more information, see our Merge Messages documentation.
You may also use the Split-Merge Block to combine outputs from Automations Writers in workspace actions. Oftentimes, a single workspace run with an Automations Writer outputs multiple messages through the Automations Writer port, which triggers downstream actions to run multiple times for one initial input message from a trigger. You often only need to send one outgoing message with an external action or process a downstream summary workspace once. In this case, the Split-Merge Block will combine output messages from Automations Writers within it and merge them into one message to output.
In this example, the Process State workspace action outputs 60 messages through the County Automations Writer port, which enter the Split-Merge Block and trigger the Process County workspace to run 60 times. Each county contains 15 blocks, so the Block Automations Writer outputs 900 messages and triggers the Process Block workspace to run 900 times. The Split-Merge Block waits for all 900 messages from the Process Block workspace, merges them by county, and outputs only 60 messages to the Downstream Work action that processes one job per county.
While the Split-Merge Block is handy for your parallel processing or Automations Writer workflows, there are some limitations you should be aware of when you use it:
Frank is designing an automation to perform asynchronous data updates daily. He has multiple workspaces for data updates and post-processing, performs data validation, and makes a final product once the updates are complete. Frank already has the workspaces on FME Flow, and the next step is to create the automation.
In this lesson, you will:
The example workspaces for this exercise are placeholders – they don't really carry out any data integration. However, the Automation design follows common real-world scenario by performing data updates and validation checks.
Frank has another data update that takes longer and can run asynchronously to the SpeedyDataUpdate workspace.
By splitting the workflow, the SpeedyDataUpdate and LongerDataUpdate workspaces will run simultaneously on different FME Engines, as long as multiple engines are available.
After the LongerDataUpdate workspace is complete, the data must undergo post-translation processing.
After both data update processes, Frank needs to run his Validation and MakeDataProduct workspaces in order.
If you start and trigger the automation, the trigger will send a message to both the Speedy and Longer Data Update workspaces and trigger them to run simultaneously. Once the Speedy Data Update finishes, it sends a message to run the Validation and Make Data Product workspaces. After the Longer Data Update is complete, the Post Processing workspace runs and sends a message to run the Validation and Make Data Product workspaces. The automation runs the Validation and Make Data Product workspaces twice: it splits the data stream into two, then combines them in the Validation workspace. 
Frank needs to run the Validation and Make Data Product workspaces only once per automation trigger, not twice. Therefore, you'll need to use the Split-Merge Block to split the workflow and merge the messages from the Data Updates before continuing with the Validation workspace.
If you are very quick, you may catch both the Speedy Data Update and the Longer Data Update workspaces running simultaneously from the Running tab. That is when both workspaces are being processed on separate FME Engines.
You've used the Split-Merge Block to orchestrate parallel processing and merge the workflow back into a single stream before continuing with downstream workspace actions. This automation accomplishes Frank's task for data updates while leveraging FME Flow's parallel processing and the Split-Merge Block.
If you're taking a live Safe Software-hosted training course or using an on-demand lab for this course, we've already imported the content for this lesson into your FME Flow.
If you're completing this lesson with your own FME Flow, you must import the starting project with the content into your FME Flow.
See FME Flow Authoring: Instructions to Import the Starting Project for directions to import the project to your own FME Flow.
In FME Flow Automations, you may split your workflows to process multiple actions simultaneously and use FME Flow's parallel processing capabilities with numerous engines. Splitting your workflow lowers the processing time by running multiple workspaces at once. It also increases fault tolerance, as workspaces that run in parallel will still process even if one workspace fails.
When you split your workflows, each stream creates an output. To combine the outputs into a single message in Automations, you use a Split-Merge Block. The Split-Merge Block lets you split the Automation workflow into multiple streams and then wait for all components within the block to complete before continuing the workflow along a single stream.
In this workflow, the trigger passes one message to the Split-Merge Block. The workflow is split and sends two messages, one to each workspace. Once the workspaces are complete, each workspace sends a message to the block's success port. The Split-Merge Block combines the two messages into one message and passes it along to the Validation workspace action. Two messages would pass to the Validation workspace without the Split-Merge Block, triggering it to run twice.
The Merge Messages action in FME Flow Automations functions similarily to the Split-Merge Block by merging messages and waiting to process all downstream actions until it receives messages from all previous actions. Compared to the Split-Merge Block, the Merge Messages action limits you from merging messages from multiple triggers and from Automation Writer outputs. Due to these, the Merge Messages action is legacy functionality in Automations and is only present to support backwards compatibility for your workflows. For more information, see our Merge Messages documentation.
You may also use the Split-Merge Block to combine outputs from Automations Writers in workspace actions. Oftentimes, a single workspace run with an Automations Writer outputs multiple messages through the Automations Writer port, which triggers downstream actions to run multiple times for one initial input message from a trigger. You often only need to send one outgoing message with an external action or process a downstream summary workspace once. In this case, the Split-Merge Block will combine output messages from Automations Writers within it and merge them into one message to output.
In this example, the Process State workspace action outputs 60 messages through the County Automations Writer port, which enter the Split-Merge Block and trigger the Process County workspace to run 60 times. Each county contains 15 blocks, so the Block Automations Writer outputs 900 messages and triggers the Process Block workspace to run 900 times. The Split-Merge Block waits for all 900 messages from the Process Block workspace, merges them by county, and outputs only 60 messages to the Downstream Work action that processes one job per county.
While the Split-Merge Block is handy for your parallel processing or Automations Writer workflows, there are some limitations you should be aware of when you use it:
Frank is designing an automation to perform asynchronous data updates daily. He has multiple workspaces for data updates and post-processing, performs data validation, and makes a final product once the updates are complete. Frank already has the workspaces on FME Flow, and the next step is to create the automation.
In this lesson, you will:
The example workspaces for this exercise are placeholders – they don't really carry out any data integration. However, the Automation design follows common real-world scenario by performing data updates and validation checks.
Frank has another data update that takes longer and can run asynchronously to the SpeedyDataUpdate workspace.
By splitting the workflow, the SpeedyDataUpdate and LongerDataUpdate workspaces will run simultaneously on different FME Engines, as long as multiple engines are available.
After the LongerDataUpdate workspace is complete, the data must undergo post-translation processing.
After both data update processes, Frank needs to run his Validation and MakeDataProduct workspaces in order.
If you start and trigger the automation, the trigger will send a message to both the Speedy and Longer Data Update workspaces and trigger them to run simultaneously. Once the Speedy Data Update finishes, it sends a message to run the Validation and Make Data Product workspaces. After the Longer Data Update is complete, the Post Processing workspace runs and sends a message to run the Validation and Make Data Product workspaces. The automation runs the Validation and Make Data Product workspaces twice: it splits the data stream into two, then combines them in the Validation workspace.
Frank needs to run the Validation and Make Data Product workspaces only once per automation trigger, not twice. Therefore, you'll need to use the Split-Merge Block to split the workflow and merge the messages from the Data Updates before continuing with the Validation workspace.
If you are very quick, you may catch both the Speedy Data Update and the Longer Data Update workspaces running simultaneously from the Running tab. That is when both workspaces are being processed on separate FME Engines.
You've used the Split-Merge Block to orchestrate parallel processing and merge the workflow back into a single stream before continuing with downstream workspace actions. This automation accomplishes Frank's task for data updates while leveraging FME Flow's parallel processing and the Split-Merge Block.
